Image rendering applications, such as Adobe Photoshop® (available from Adobe Systems Incorporated, San Jose, Calif.), are widely popular and enable different image-related operations. Examples of such operations include image warping and feature recognition.
Image warping is a method of modifying an image by changing the spatial configuration of portions of the image. A source image can be warped into a destination image according to a specified mapping between the source image and the destination image. For instance, an object in the source image can be resized and moved to a new location in the destination image.
In comparison, feature recognition is a method of recognizing a feature rendered in an image from properties of the image pixels. For instance, facial recognition software can identify facial features (mouth, nose, eyes, etc.) of a person.
Generally, an image rendering application can provide both image warping and feature recognition operations. To do so, a feature is first recognized and marked in the source image. Subsequently, the image rendering application finds the new location of the feature in the destination image based on the mapping between the two images. The new location is marked as that of the feature.
Various technical challenges, including computational efficiency and accuracy, arise when the image rendering application is configured to provide both operations. As a solution, existing systems generally trade-off between computational efficiency and accuracy.
For example, an existing system configures an image rendering application to use a one-to-one pixel mapping between source and destination images. This mapping tracks each pixel in the two images, thereby achieving high accuracy with respect to locating a feature in the destination image. However, because each pixel is tracked, the computational burden (e.g., processor and memory usage) is relatively high. The high computational burden can translate into user interface (UI) lag, thereby degrading the user experience.
In another example, an existing system configures an image rendering application to use meshes. A mesh is generally sparse and has a lower resolution than an image. An example of the mesh is a subset of pixels from the image (e.g., every fourth pixel in the image). Rather than mapping source and destination images, the image rendering application uses source and destination meshes that correspond to the two images. Generally, the lower the resolution, the higher computational savings are (e.g., for a 25% resolution—every fourth pixel—the computational savings can be up to 93.75%).
However, source and destination meshes provide inaccurate results when used for locating, in the destination image, a feature from the source image. One reason for this inaccuracy involves displacement offsets. Specifically, the image warping can displace a feature, where the displacement is at an offset relative to resolution of the destination mesh. For instance, the resolution of a mesh as compared to an image is every fourth pixel, but an eye (or some other facial feature) may be displaced by three pixels in an image warping operation. If the mapping between the meshes does not properly track the offset (e.g., one pixel in the illustrative eye example), the image rendering application incorrectly tracks the feature's location in the destination image. Practically, rather than locating the eye in the destination image, the image rendering application incorrectly identifies an eyebrow, thereby also degrading the user experience.
Another reason for the feature-location inaccuracy are holes in the mapping. Specifically, a warping operation can compress the source image. In certain situations, the compression results in points on the source mesh not having neighboring vectors for mapping these points to points in the destination image. When averaging is applied for the mapping, the absence of neighboring vectors causes the averaging techniques to fail, which results in the holes. Hence, if a feature falls in a hole, that feature can no longer be located in the destination image. Practically, if the eye is recognized in the source image but is unmapped, the image rendering application no longer identifies the eye in the destination image because of the holes in the mapping. This also degrades the user experience.